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INTRODUCTION 


Psychologically-based  treatments  and  cognitive  behavioral  therapy  (CBT)  interventions  have 
been  shown  to  be  effective  in  alleviating  symptoms  of  Post-Traumatic  Stress  Disorder  (PTSD) 
and  related  psychological  health  difficulties  in  Veterans  and  military  personnel  who  suffer  from 
these  problems.  To  meet  the  increased  service  needs  of  Veterans  with  PTSD,  new  training 
methods  need  to  be  developed  which  are:  1)  evidence-based,  and  2)  effective  in  modifying  and 
sustaining  changes  in  provider  behavior.  Methods  of  training/implementation  must  also  be 
scalable,  and  feasible  for  delivery  to  large  numbers  of  providers  in  cost-effective  ways. 
Internet-based  training  is  a  promising  new  approach  for  meeting  this  need,  but  has  received 
little  systematic  evaluation  to  date.  Noting  the  urgency  and  high  priority  of  this  issue,  Fairburn 
and  Cooper  (2011)  have  advocated  strongly  for  the  development  of  novel,  internet-based 
training  methods  and  innovative  research  designs  to  test  the  effectiveness  of  these  new  training 
methods.  Our  current  program  of  research  is  aimed  to  address  these  needs. 


The  broad  objective  of  our  research  is  to  design,  implement  and  evaluate  scalable  and  cost- 
effective  new  methods  for  training  of  mental  health  clinicians  providing  treatment  services  to 
veterans  with  PTSD.  The  randomized  controlled  trial  (RCT)  design  is  briefly  as  follows:  eligible 
clinicians  in  the  community  and  VPIA  will  be  randomly  assigned  in  equal  numbers  to  three 
parallel  intervention  condition:  a)  Web-based  training  plus  web-centered  supervision;  b)  Web- 
based  training  alone;  and  c)  Training-as-usual  control  group.  An  equal  number  of  clinician 
trainees  from  VPIA  (N=219)  and  the  community  (N=219)  will  be  recruited  and  enrolled  in  the 
study  over  an  18-month  period  according  to  a  randomized,  stratified  24-week  design. 
Comprehensive  assessments  will  be  performed  at  baseline  (TO),  completion  of  training  (T1), 
and  at  3  month  follow-up  (T2).  Participants  randomized  to  the  consultation  condition  will  be 
exposed  to  a  newly  developed  web-centered  form  of  learning  consultation.  Measures  of 
compliance  and  completion  will  assess  adherence  to  protocol.  Training  effectiveness  will  be 
evaluated  by  means  of  a  combination  of  objective  (SPE)  and  self-report  measures. 


The  primary  and  secondary  aims  of  the  study  are  as  follows: 

Primary  Aim:  To  compare  an  enhanced,  internet-based  training  intervention  combined  with 
novel  web-centered  supervision,  internet-based  training  intervention  without  web-centered 
supervision  and  a  wait-list  control  with  regard  to  improvements  in  two  CBT-based  skill  areas 
(behavioral  task  assignment  and  case  conceptualization).  We  hypothesize  that  enhanced, 
internet-based  training  in  conjunction  with  web-centered  supervision  will  result  in  superior  skills 
acquisition  compared  to  internet  training  alone  and  that  internet  training  alone  will  result  in 
superior  CBT  skills  than  wait-list  control. 

Secondary  Aim  #1:  To  compare  improvements  in  knowledge  and  attitudes  following  internet- 
based  training  with  or  without  web-centered  supervision  and  the  control.  We  hypothesize  that 
web-centered  supervision  will  lead  to  greater  improvements  in  CBT  knowledge  and  perceived 
self-efficacy  compared  to  internet-based  training  without  supervision  or  a  written  training-only 
condition.  We  hypothesize  similarly  that  internet-based  training  will  be  associated  with  improved 
outcomes  in  CBT  knowledge  and  attitudes  compared  to  a  written  training-only  condition. 

Secondary  Aim  #2:  To  compare  improvements  in  skills  acquisition  in  knowledge  and  attitudes 
following  training  in  clinicians  recruited  from  VFIA  mental  health  treatment  settings  compared  to 
those  providing  services  in  civilian  community-based  clinics.  We  hypothesize  that  comparable 
improvements  will  be  achieved  in  the  trainees  from  civilian  community-based  clinics  compared 
to  clinicians  recruited  from  VFIA  centers. 


Prepared  by  New  England  Research  Institutes,  Inc 

480  Pleasant  Street,  Watertown,  MA  02472  USA 
Tel:  617.923.7747—  Fax:  617.926-0144 
Page  4  of  1 1 


Secondary  Aim  #3:  To  assess  the  relative  efficiency  of  training,  as  measured  by  total  time 
required  for  training  in  each  condition,  in  addition  to  self-reported  level  of  burden  for  clinicians. 
We  hypothesize  that  internet-based  training  with  or  without  web-centered  supervision  will  be 
associated  with  increased  time  investment  and  burden  relative  to  training-as-usual,  but  that 
absolute  levels  of  burden  will  be  low  in  the  web  training  conditions. 


This  study  will  be  the  first  of  its  kind  to  systematically  compare  web-based  training  interventions 
across  treatment  settings  and  provider  groups  (VHA  vs  Non-VHA).The  study  will  also  be  unique 
in:  1)  developing  and  testing  of  new  web-enhanced  training  modules  and  a  novel  web-centered 
supervision  model  recently  proposed  by  Fairburn  &  Cooper  (2011);  2)  development  and 
implementation  of  a  new  patient-reported  measure  of  clinician  skill  and  competency;  and  3) 
assessment  of  post-training  maintenance  of  skills  beyond  the  training  period.  Our  focus  on 
broad-based,  generic  CBT  skills  rather  than  more  narrowly  focused  protocol-based  skills  is 
another  innovative  aspect  of  our  proposed  study.  Finally,  the  use  of  standardized  patient 
methodology  for  assessing  outcomes  of  training,  and  planned  comparisons  with  self-report  and 
knowledge-based  assessment,  is  another  novel  feature  of  our  proposed  study. 


If  successful,  the  study  will  promote  a  better  standard  of  care  for  psychological  health  of 
Veterans  and  their  families  by  evaluating  technical  feasibility  of  two  training  models  in  evidence- 
based  skills  for  PTSD  treatment  providers  and  measuring  their  outcomes  and  effectiveness.  If 
successful,  the  study  will  provide  experimental  support  for  broad  implementation  of  these 
enhanced  new  training  methods  across  a  variety  of  treatment  settings. 


BODY 

This  section  of  the  report  will  describe  the  research  accomplishments  associated  with  each  task 
outlined  in  the  approved  Statement  of  Work  (SOW).  As  stated  in  our  SOW,  the  major  tasks  to 
be  accomplished  in  year  3  were: 

Develop  Data  Management  System 

A.  Ongoing  monitoring  of  study  process 

i.  Between  years  2  and  3  of  this  project,  the  team  has  actively  monitored  data 
collection,  data  management,  and  the  study  process.  We  have  developed  bi¬ 
weekly  reports  for  active  tracking  of  recruitment  and  study  processes,  and  meet 
bi-weekly  to  discuss  progress.  Study  enrollment  was  completed  in  April  2015  with 
a  total  of  420  participants  (VA  =  209,  Community  =  211). 


Develop  Standardized  Patient  Rating  Protocol 

A.  Develop  and  Pre  Test  Standardized  Patient  (SP)  Hating  Guide 

i.  The  team  has  developed  the  SP  Rating  Guide  in  year  2.  During  year  3,  rating 
scales  have  been  developed  and  finalized. 

B.  Standardized  Patient  (SP)  Ratings 

i.  During  year  3,  raters  have  been  utilizing  the  finalized  rating  forms  to  rate  the  SPs 
from  BTA  and  CA.  These  ratings  have  undergone  and  continue  to  undergo 
analyses  for  inter-rater  reliability. 

C.  Training  of  Independent  Study  Raters  to  Concordance 
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i.  The  team  has  hired  and  trained  study  raters  for  BTA  and  CA.  Blinded 
independent  ratings  and  reliability  assessents  began  in  Q2  of  Year  2. 


Develop  Data  Collection,  Intervention  and  Standardized  Patient  Interviews 

A.  Web  Screening  of  Potential  Clinician  Applicants 

B.  Schedule  and  Carry  Out  SP  Baseline  Interviews  with  Eligible  Clinicians 

C.  Automated  Random  Assignment  of  Participant  with  Completed  SP  Interview  by  Web 
Program  to  Web-Course,  Web-Course  +  Web-  Based  Supervision,  or  Control 

D.  Participants  Complete  Assigned  Condition 

i.  Web  course  participants 

ii.  Web  course  +  web-based  supervision 

iii.  Control  Condition 

E.  Schedule  and  Carry  Out  Post  Training,  Follow  Up  SP  Interviews 

i.  This  is  an  ongoing  occurrence  into  year  3.  Completion  is  scheduled  for  Q4  201 5. 

Current  recruitment  information  is  as  follows  (as  of  10/13/2015): 

Registration  Statistics 

1609  registrant 

Screener  Statistics 

Out  of  1 609  registrants.. . 

998  participants  have  screened  eligible 
68  participants  have  screened  ineligible 
543  participants  have  not  screened 

Consent  Statistics 

Out  of  998  participants  screened  eligible... 

856  participants  have  consented 
64  participants  have  not  consented 
78  participants  have  not  completed  the  consent 

Pre-test  Statistics 

Out  of  856  participants  consented... 

613  participants  have  completed  the  pre-test 
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Standardized  Patient  1  Statistics 

Out  of  613  participants  who  completed  the  pre-test. . . 
420  participants  have  completed  the  SP1 


Randomization  Statistics 

Out  of  420  participants  who  completed  the  SP1 ... 

420  participants  have  been  randomized 

66  participants  have  been  randomized  but  have  decided  to  not  partake  in  the  study  (early 
termination) 

Post-test/SP2  Statistics 

Out  of  4 20  participants  who  were  randomized... 

273  participants  have  completed  the  SP2 
241  participants  have  completed  the  post-test 

Follow-up  Statistics 

Out  of  the  420  participants  who  have  been  randomized  ... 

224  participants  have  completed  the  Follow-up 

Standardized  Patient  3  Statistics 

Out  of  420  participants  who  have  been  randomized. . . 

244  participants  have  completed  the  SP3 


Analysis  and  Evaluation 

A.  Monitor  Data  Collection  Rates  and  Data  Quality 

i.  The  team  continues  to  collaborate  with  the  VPIA  NCPTSD  to  monitor  the  collection 
rates  and  data  quality  on  a  weekly  basis. 

B.  Create  Interim  and  Final  Analytical  Data  Sets 

i.  During  the  second  year  of  this  project,  the  initiation  of  the  data  sets  was  scheduled 
to  begin.  This  task  will  continue  to  occur  into  year  4.  The  team  has  initiated 
discussion  on  what  the  data  sets  should  consist  of,  and  has  begun  weekly 
scheduled  discussions  on  data  cleaning  and  analysis  plans.  A  statistical  analysis 
plan  is  in  the  process  of  being  developed. 
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KEY  RESEARCH  ACCOMPLISHMENTS 


•  Web-based  and  written  training  materials  were  finalized 

•  Video  elements  were  implemented  into  the  training  website 

•  Training  website  was  launched,  to  include  Behavioral  Task  Assignment  &  Chain 
Analysis  and  Case  Formulation 

•  CE  accreditation  has  been  obtained  for  Psychology,  Social  Work,  Nursing,  and  CME 

•  Standardized  Patient  Interview  scripts  were  finalized,  and  actors  were  hired  and  trained 

•  Pre-test,  post-test,  and  follow-up  questionnaires  were  developed,  finalized,  and 
programmed 

•  All  data  collection  forms  were  finalized,  programmed,  and  tested  in  the  data 
management  system 

•  Reports  to  review  ongoing  statuses  of  all  forms  and  subject  progress  were  developed 
and  finalized 

•  Subject  recruitment  methods  were  finalized  and  recruitment  was  started  (ongoing) 

•  Subject  management  procedures  for  the  entire  study  flow,  including  all  email  and  phone 
communications  were  developed  and  are  ongoing 

•  IRB  approval  has  been  sought  and  obtained  for  all  necessary  modifications,  and 
continuing  review  approval  was  obtained 

•  The  study  was  launched,  and  has  so  far  randomized  1 94  of  41 4  participants  (as  of 
September  17th,  2014).  All  participants  are  anticipated  to  be  randomized  by  Q2  of  Year 

3. 

•  Recruitment  goals  were  exceeded  with  a  final  randomization  number  of  N  =  420. 


During  the  upcoming  performance  period,  year  4,  the  team  will  focus  on  the  following  activities 
and  milestones: 

1 .  Ongoing  monitoring  of  study  progress  -  We  will  monitor  the  last  remaining  participants  to 
complete  data  collection  at  all  three  timepoints  of  the  study. 

2.  Perform  blinded  independent  ratings  and  reliability  assessments  -  We  will  continue 
blinded  independent  ratings  and  ongoing  monitoring  of  inter-rater  reliability. 

3.  Post-Training  Assessment  Review  and  Follow-Up  -  Based  on  the  established 
recruitment  plan,  all  post-test  and  follow-up  questionnaires  will  be  completed  in  October 
2015. 

4.  Create  interim  and  final  analytic  data  sets  -  Data  sets  are  being  finalized  and  details  foot 
the  statistical  analysis  plan  (SAP).  This  task  will  continue  into  Year  4. 

5.  Perform  all  analyses  according  to  specification  -  During  Year  4,  the  team  will  finalize 
SAPs  for  each  proposal  and  perform  statistical  analyses  to  address  study  aims  (above). 

6.  Author  and  co-author  evaluation  findings  -  During  Year  4,  findings  will  be  analyzed  and 
discussed  in  developed  manuscripts.  We  anticipate  submitted  3-4  papers  prior  to  the 
termination  of  the  grant. 
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7.  Convene  advisory  meeting  and  project  review  -  During  Year  4,  the  team  will  review 
study  achievements  and  lessons  learned. 


To  date  there  have  been  no  risks  or  unanticipated  issues  associated  with  this  project  that  have 
impeded  its  performance. 


Personnel  receiving  salary  from  this  research  effort  are  Raymond  C.  Rosen,  Ph.D.  (Partnering 
PI),  Lisa  Marceau,  MPH  (Co  Investigator,  Director),  Ashley  Wilkinson  (Project  Manager),  Gayatri 
Ranganathan,  MS  (Statistician),  Bernet  Kato  (Statistician),  Julia  Coleman  (Research  Associate), 
Julia  Dwyer  (Research  Assistant),  Shreya  Divatia  (Data  Manager). 


REPORTABLE  OUTCOMES 

Dr.  Rosen  presented  an  update  on  the  project  to  CDMRP  leadership  at  the  MOMRP  meeting  in 
Ft.  Detrick  on  September  11, 2015.  The  project  update  was  well  received  and  no  concerns 
were  raised  regarding  timelines  or  completion  of  the  project. 


All  presentations  to  date: 


Title 

Presenter 

Conference  Dates 

A  Randomized  Control  Trial  of  Online  Training 
for  PTSD 

Alie,  G.  &  Graham,  B.C. 

April  2013 

Actor  Fidelity  and  Training  for  Standarized 

Patient  use  in  Clinical  Research 

Jordan,  K.R.  &  Graham, 

B.C. 

April  2014 

Web-centered  Consultation  for  Online  Training 
in  PTSD  Treatment:  A  Scalable  Approach  to 
Skill-building 

Sardarian,  S.  &  Graham, 

B.C. 

April  2014 

Individual  and  Organizational  Factors  in 
Dissemination  and  Implementation  of  Skills 
following  an  Online  Training  for  Clinicians 
Treating  PTSD 

Graham,  B.C.,  Ruzek,  J.  & 
Lee,  J.  E. 

May  1  -  May  3,  2014 

Randomized,  Controlled  Trial  of  CBT  Training 
for  PTSD  Providers:  Project  OUTFIT 

Ruzek,  J.  &  Rosen,  R. 

August  5  -  August  6, 

2014 

Project  OUTFIT :  Online  User  T raining  for 
Intervention  in  Trauma 

Graham,  B.C.  &  Ruzek,  J. 

April  28,  2015 

The  Role  of  Organizational  and  Provider 

Factors  in  Community  Interventions  Across 
Diverse  Systems. 

Graham,  B.C. 

June  1,  2015 

Randomized,  Controlled  Trial  of  CBT  Training 
for  PTSD  Providers:  Project  OUTFIT 

R.  Rosen 

September  11,  2015 

Improving  the  Efficiency  of  Standardized 

Patient  Assessment  of  Clinician  Fidelity:  A 

Graham,  B.C. 

September  25  - 
Septmber  26,  2015 
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Comparison  of  Automated  Actor-based  and 
Manual  Clinician-based  Ratings _ 


CONCLUSION 

This  training  program  will  focus  on  development  of  evidence-based  CBT  skills  to  improve  skills 
in  providers  treating  Veterans  and  active  duty  military  and  to  effectively  engage  patients  in  the 
treatment  process.  This  innovative  study  will  add  new  knowledge  to  our  understanding  of  skills 
dissemination  in  PTSD  provider  care.  We  will  test  the  hypotheses  of  the  study  in  a  rigorous, 
experimental  design,  and  will  assess  outcomes  of  new  web-based  training  modules  and 
consultation  methods.  This  study  will  provide  data  to  assist  researchers,  military  leaders,  and 
treatment  providers  to  better  understand  practical  and  theoretical  implications  for  future  training 
of  mental  health  providers  in  the  VAH  and  other  health  systems. 


All  major  milestones  were  met  in  the  past  12  months  of  grant  activity. 
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DMRDP 


Study/Product  Aim(s) 

The  purpose  of  this  study  is  to  design,  implement  and  evaluate  a  web  based  training 

program  providing  skills-oriented  continuing  education  for  mental  health  professionals. 

•  To  compare  improvements  in  knowledge  and  attitudes  following  internet-based 
training  with  or  without  web-centered  supervision  and  the  control. 

•  To  compare  improvements  in  skills  acquisition  in  knowledge  and  attitudes  following 
training  in  clinicians  recruited  from  VHA  mental  health  treatment  settings  compared 
to  those  providing  services  in  civilian  community-based  clinics. 

•  To  determine  whether  clinician  implementation  of  skills  assessed  by  means  of  a  novel, 
patient-based  measure  of  clinician  skills  implementation  and  effectiveness  is 
predictive  of  changes  in  an  objective  (i.e.,  standardized  patient)  measure  of  skills 

•  To  assess  the  relative  efficiency  of  training,  as  measured  by  total  time  required  for 
training  in  each  condition,  in  addition  to  self-reported  level  of  burden  for  clinicians. 


The  objective  is  to  learn  whether  novel,  internet- 
based  training  methods,  with  or  without  web- 
centered  supervision,  may  provide  an  effective 
means  to  train  increasing  numbers  of  mental 
health  providers  in  relevant,  evidence-based 
clinical  skills. 


Approach 

The  study  will  test  a  large-scale,  Web-based  method  of  training  with  414  VA  (n  =  207) 
and  community-based  clinicians  (n  =  207).  Subjects  will  be  recruited  to  participate  in  the 
evidence-based  skills  training  and  will  be  randomly  assigned  to  1  of  3  groups:  Interactive 
Web-based  training  only;  interactive  Web-based  training  plus  post  training  Web-based 
consultation;  or  training  as  usual  control.  Effectiveness  will  be  evaluated  using  an  intent- 
to-train  design.  Post-training  and  3-month  follow  up  assessment  will  be  conducted. 


Accomplishments:  Monitoring  of  data  collection,  data  management  and  study 
processes  continues..  Recruitment  goals  were  exceeded  by  enrolled  209  VA 
participants  and  211  Community  participants.  Standardized  Patient  Rating  are  in  the 
process  of  being  conducted.  Data  collection  will  be  completed  in  Q4  2015. 


Timeline  and  Cost 


Activities 


Develop  Web-based  Training  Materials 
Develop,  Pre-Test  and  Finalize  Web  Site 
including  Audio  and  video 

Develop  /  Finalize  Study  Protocol 
Measures 

Develop  Data  Management  System 
Prepare  SP  Rating  Protocol 
Data  collect.  Intervention,  SP  Interviews 
Analyses  and  manuscripts 
Estimated  Budget 


Goals/Milestones 

CY12  Goal  -  Develop  SP  and  Consultation  protocols,  design  web  program, 
develop  DM  System 

□  Content  completed 

□  Video/audio  created 

□  SP  scripts  developed 

□  Site  programmed  and  tested 

□  DM  system  developed 
CY13  Goals-  Data  collection 

□Track  metrics  for  subject  recruitment  and  completion  goals 
CY14  Goals-  Data  collection 

□Track  metrics  for  subject  recruitment  and  completion  goals 
CY15  Goals/CY16  Goals  -  Analyses  and  manuscripts 

□  Complete  main  results  analyses  and  MS  based  on  findings 

Comments/Challenges/lssues/Concerns 

Currently  slightly  over  budget.  For  Y04,  labor  hours  are  being  carefully 
monitored 

Projected  expenditure  - 1,057,457 

Actual  Expenditure:  $1,078,948 
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